Skip to content

feat(entity,storage): rework speculation tree model and store#231

Draft
behinddwalls wants to merge 1 commit into
mainfrom
preetam/ext/speculation-store
Draft

feat(entity,storage): rework speculation tree model and store#231
behinddwalls wants to merge 1 commit into
mainfrom
preetam/ext/speculation-store

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

The speculation design (doc/rfc/submitqueue/speculation.md, merged in #230) makes a path's score a dynamic, scorer-owned prediction — recomputed on every respeculate as dependencies land, dependency builds pass, and siblings fail — not a fixed number stamped at enumeration. The prior store model drifted from that in three ways: it documented Score as the enumerator's, it carried a Score on the Build entity that was never populated or read, and it left the MySQL column name out of step with the renamed entity field. This aligns the entity + store with the design.

What?

entity: SpeculationPath{Base, Head} is the unit; SpeculationPathInfo carries the path plus its Score, controller-owned Status (candidate/selected/building/passed/failed/cancelled), and BuildID. Score is now documented as scorer-computed and controller-persisted dynamic state — recomputed on every respeculate, not set at enumeration. Adds SpeculationPathAction and SpeculationPathDecision for the selector seam. SpeculationTree.Speculations becomes Paths. The Build entity uses the shared SpeculationPath (Head = the batch under verification) and drops its own Score, which was never populated or read; the prioritizer seam re-adds a build priority when it lands.

storage: SpeculationTreeStore.UpdateSpeculations(batchID, []SpeculationInfo) becomes Update(ctx, SpeculationTree) — symmetric with Create, keyed by tree.BatchID. The MySQL column speculations is renamed paths to match the entity field. MySQL impl, mock, and storage integration coverage added/updated.

Test Plan

  • make test — 65/65 pass
  • ✅ storage integration test (MySQL via docker-compose) — new speculation-tree cases: create/get round-trip preserving Base/Head/Score/Status/BuildID, duplicate → ErrAlreadyExists, whole-tree overwrite via Update, and Get/Update on a missing batch → ErrNotFound
  • make gazelle / make fmt — no drift

@behinddwalls behinddwalls force-pushed the preetam/ext/speculation-store branch from 927e240 to 6bc8426 Compare June 10, 2026 03:45
@behinddwalls behinddwalls force-pushed the preetam/ext/speculation-rfc branch from 380f0b1 to 3eddc3d Compare June 10, 2026 03:45
@behinddwalls behinddwalls force-pushed the preetam/ext/speculation-store branch from 6bc8426 to 7b49586 Compare June 11, 2026 03:27
@behinddwalls behinddwalls force-pushed the preetam/ext/speculation-rfc branch from 67da1e9 to 0ee54c9 Compare June 11, 2026 03:28
@behinddwalls behinddwalls force-pushed the preetam/ext/speculation-rfc branch 5 times, most recently from 3053dd4 to 58cabeb Compare July 7, 2026 22:43
behinddwalls added a commit that referenced this pull request Jul 7, 2026
## Summary

Capture how SubmitQueue speculates before the interfaces land.
Speculation is modeled as two layers: **decision seams** — enumeration
and **scoring** *describe* the tree of possible bets, selection and
prioritization *act* on it — and **signal-driven limit policies** that
decide *how much* to allow at each step (dependency limit, selection
limit, prioritization limit), scaling with the build system's resources.
A path's score is dynamic — recomputed as dependencies land, fail, or
have their builds pass — not frozen at enumeration.

Covers: the problem (dependent batches serialize behind multi-minute
builds), the Base/Head path-and-tree model, where speculation sits in
the orchestrator pipeline, the status-vs-action ownership split, the
dependency limit as an eligibility gate (batches wait for predecessors
to land rather than trimming their base), dynamic rescoring, and how
prioritization rations the queue's builds against its budget.

Design only; entities, store, and extension interfaces follow in later
PRs in this stack. Adds doc/rfc/submitqueue/speculation.md and links it
from the RFC index.

### Changes since review

Reframed from two seams (enumeration + selection) to describe/act
decisions + limits, addressing @sbalabanov's review:
- selector resource-awareness → the new **prioritization** layer owns
the queue-wide build budget; the selector stays a pure per-batch policy
- "variable depth, not a single knob" → the **dependency limit** is a
signal-driven policy, not a constant
- clarified the dependency limit is an **eligibility gate** on active
in-flight dependencies (a batch waits until predecessors land), not a
base trim
- **dynamic scoring**: split scoring out of enumeration into a
**scorer** that recomputes each path's score as dependencies
land/fail/pass — enumeration is now structure-only
- entire tree persisted (the selector reads all path statuses + scores,
so it can cancel a path it earlier asked to build); the redundant
fallback path is cancelled on a winning bet; slash-free example names;
trimmed impl-detail bullets
- recorded as open: extending rescoring to weigh how long a batch has
waited

## Test Plan

Doc-only. Rendered locally; ASCII diagrams and tables verified; no
hard-wrapped prose.

## Issues


## Stack
1. @ #230
1. #231
1. #232
Reshape the speculation tree data model around the Base/Head path that the build stage actually consumes, and align its store with the speculation design.

entity: SpeculationPath{Base, Head} becomes the unit; SpeculationPathInfo carries the path plus its Score, controller-owned Status (candidate/selected/building/passed/failed/cancelled), and BuildID. Score is scorer-computed and controller-persisted dynamic state — recomputed on every respeculate, not set at enumeration. Adds SpeculationPathAction and SpeculationPathDecision for the selector seam. SpeculationTree.Speculations becomes Paths. The Build entity uses the shared SpeculationPath (Head = the batch under verification) and drops its own Score, which was never populated or read; the prioritizer seam re-adds a build priority when it lands.

storage: SpeculationTreeStore.UpdateSpeculations(batchID, []SpeculationInfo) becomes Update(ctx, SpeculationTree) — symmetric with Create, keyed by tree.BatchID. The MySQL column speculations is renamed paths to match the entity field. MySQL impl, mock, and storage integration coverage (create/get round-trip, duplicate, whole-tree overwrite, not-found) added/updated.
@behinddwalls behinddwalls force-pushed the preetam/ext/speculation-store branch from 7b49586 to 4606cc7 Compare July 7, 2026 23:18
@behinddwalls behinddwalls changed the base branch from preetam/ext/speculation-rfc to main July 7, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant